In recent years, more and more multimedia contents were used on social media to share users' experiences and emotions. Compared to single text or image, the complementation of text and image can further fully reveal the real emotion of users. Concerning the sentiment shortage of single text or image, a method based on Convolutional Neural Network (CNN) was proposed for multimedia sentiment analysis. In order to explore the influence of semantic representation in different level, image features were combined with different level (word-level, phrase-level and sentence-level) text features to construct CNN. The experimental results on two real-world datasets demonstrate that the proposed method gets more accurate prediction on multimedia sentiment analysis by capturing the internal relations between text and image.